Working on comments for reaper/poller PRs.
Rucio meeting. Coral sprint review. PUNCHLUNCH talk. Some notes:
For SKAO, like many communities, metadata is an extremely important consideration in our data management solution where we need to, for example
1) trace the provenance of files, so where they've came from
2) track how files may have been processed, that is what level of processing a particular version of a file has had done to it, and
3) to facilitate search capabilities
important to note that we're very much in an exploratory phase here. Rucio isn't the only storage technology that SKAO is prototyping - we're currently gathering potential use cases and assessing feasibility.
moving on to so a small update about what's been made possible in the past few releases. and this is work that SKAO has been directly involved in contributing to having recognised some obvious gaps in functionality
--- slides
1.
up until v1.27: so you could have key1=value1, key2=value2
filter engine supports most operators that you'd expect for search operations
in our example: rucio list-dids-extended is the command to list files in the database using metadata from the custom metadata store. with the filter argument here, we are filtering on test_key between 1 and 3 (with AND being represented by a ,) OR test_key_2 greater than some date, in this case the first of june 2022.
2.
rucio uses a plugin system: query metadata is list_dids method
MongoDB is a nosql document store.
TAP access: e.g. store RA and DEC, celestial coordinates in your metadata, and perform simple cone searches, and also access via TAP
3.
SKAO test: we work within a project framework called SAFE which plans work on a 3 monthly cadence. This work has not been officially planned for this current increment, so it is unlikely to happen in the next 3 months
rucio-analysis: these tests this will be made part of the rucio-analysis framework. This framework was written in python3 for the ESCAPE project as a way of encouraging code reusability when writing capability tests, the results of which would later be used in project deliverablesRucio knowledge share w/ blue-lavender and coral
Making PRs for reaper OIDC bug and missing oidc_account in poller.
Successful replication from IMPERIAL to STFC_STORM, in short we needed to:
Finished slides for tomorrow's rucio knowledge share.
Tokens debugging:
In this workflow, the token retrieved at authentication by the user using the IAM auth client (auth_code flow) is not used. Instead, Rucio server gets a token via the IAM admin client using the client_credentials grant. This type of grant is for app-app communication. It doesn’t typically require the offline_access scope (as this scope is needed to generate refresh tokens, but refresh tokens aren’t required when the app has all the credentials it needs to generate access tokens whenever it wants). It also doesn’t make sense to have the openid scope, or groups, as these are related to users, which this type of grant doesn’t necessitate.
Knowing this, it is still possible to include the openid and offline_access scopes, although it’s ambiguous what this actually means. In fact, it is actually necessary to have the offline_access scope if you want to use the token from this flow in FTS as when a token is sent to FTS, the first thing it does is a token-exchange against its own client, hardcoding a request for this scope:
Aside: It is not necessary to include the offline_access scope in the original token request when authenticating with Rucio. It is necessary that the original client has it enabled, not that they are in the original token:
The token-exchange done by FTS exchanges an access token for a refresh token (again, quite what this means I don’t know - this token was originally a client_credentials token, it doesn’t make sense to have a refresh token). Although this exchange preserves the scopes of the original token (but adds offline_access), it trashes the audience claim if there are wlcg.groups present in the token:
If there are, it sets the audience to https://wlcg.cern.ch/jwt/v1/any, otherwise None.
Subsequently, if the audience is None, it later changes this to the FTS client ID:
So all information about the original audience is lost.
Now, as mentioned above, a client_credentials grant is not tied to a user, so having groups set doesn’t make sense. It follows from above that the audience in this workflow will always be the FTS client ID.
Here comes the rub. dCache (supposedly, from Rizart via Paul M.) needs wlcg.groups for permissions ergo this flow doesn’t work with dCache. Unsure about how other storage managers handle this, but they need to: a) not care about the audience (or we give them the FTS client ID as the audience), and b) not care about wlcg.groups for permissions.
Debugging admin token workflow on Rucio dev instance. Seems to be related to admin token from client_credentials grant having openid stripped, and FTS requesting a new refresh token via token-exchange with a hardcoded set of scopes, including openid. Indigo IAM doesn't allow a scope to be added that isn't in the subset requested by the original client where the token came from.
Can directly curl token endpoint to get a token for a client with client_credentials by:
curl -s -u <client_id>:<client_password> -d grant_type=client_credentials -d audience=https://wlcg.cern.ch/jwt/v1/any -d scope="profile wlcg.groups" https://iam-escape.cloud.cnaf.infn.it/tokenand for token exchanges with something like:
curl -s -u <client_id>:<client_password> -d grant_type=urn:ietf:params:oauth:grant-type:token-exchange -d subject_token=$TOKEN -d audience=fts -d scope="profile wlcg.groups openid" https://iam-escape.cloud.cnaf.infn.it/tokenSwitched to admin token flow, doesn't seem to work either. Spoke with Rizart and Mihai.
Metadata presentation at Rucio meeting. Slide notes:
1.
THE FIRST BIT OF WORK DONE WAS ON CREATING A FILTERING ENGINE
A quick recap.
base metadata store (referred to as column based)
json metadata store (referred to as json)
HEP specific workflows, or file metadata liek checksums, or just generally keypairs that Rucio needs for internal use
- up until 1.27.0, you could do a simple equality statement e.g. key1= value=1, and you could AND these statements together
- filtering engine, credit to Gabriele Fronze's for doing the groundwork on this
2.
list-dids extended, which ties into the metadata plugin system, allowing for custom json metadata to be used
VID:
first thing i demonstrate is string equivalence, which existed pre 1.27.0, but now supports not equals and wildcards
- so we set a string key-value pair on a file, and list-dids using a simple equivalence expression to return this file
- as we expect, the inverse not equals expression returns nothing
- add some more metadata with the same key but different value to two more files
- and then prove that the inverse expression is returning the two files with this different value
- testing the LIKE operator. this should return all three files
- and likewise, the NOT LIKE operator returns none
- moving on to date ranges, first we set a key on two separate files with dates 06-01 and 06-02
- we give it a greater than or equals to 06-01 filter, which should return the two files
- we add in an AND clause (the comma) restricting it to less than 06-02, returns a single file
- then set another file's metadata with the same key to 06-03, and add an OR clause (semicolon) to capture the first and third files, which it does
- finally, number ranges. we set number key-value pairs on three separate files, ranging from 1.5 to 3.5
- do list-dids searching for this key between 1 and 3, returns two files (missing the third)
- then finally, for good measure, add in an ORed date clause >= 06-01 (which should return all files)
3.
THE SECOND BIT OF WORK DONE WAS ON PLUGINS
postgres - can write to a json blob inside that table
storage metadata compose file, which brings up both a mongodb and a separate postgres instance to play around with
4.
5629, which is the work being done on the postgres plugin (and for which there is a pull request waiting)
5484, which is work that Cedric is doing / has done on archiving JSON metadata when a did is deleted (currently obase, column data is stored in the deleted_dids table)
We work with a framework called SAFE which plans work on a 3 monthly cadence. FTS client on IAM broken (adding scopes has reset something) - Mihai investigating.
Finished Rucio metadata SIG presentation.
Spoke with Mihai - need to limit scopes being presented to FTS (fts client can't refresh if it doesn't have permission on the scopes). Openid, profile, offline_access and wlcg.groups are all OK.
Upgraded Rucio dev server and daemons to 1.28.6. Started testing manual flow (required new client container to get rid of SSL errors) but got stuck with ESCAPE tokens not being recognised by FTS ("bad credentials"). Emailed Mihai.
Looked at Rucio 101 slides from Rohini.
Started writing Rucio metadata SIG presentation.
Implementing a postgres extension for Rucio.